Real-time Detection of Finger Picking Musical Structures

MIDIME is a software architecture that houses improvisational agents that react to MIDI messages from a finger-picked guitar. They operate in a pipeline whose first stage converts MIDI messages to a map of the state of instrument strings over time, and whose second stage selects rhythmic, modal, chordal, and melodic interpretations from the superposition of interpretations latent in the first stage. These interpretations are nondeterministic, not because of any arbitrary injection of randomness by an algorithm, but because guitar playing is nondeterministic. Variations in timing, tuning, picking intensity, string damping, and accidental or intensional grace notes can affect the selections of this second stage. The selections open to the second stage, as well as the third stage that matches second stage selections to a stored library of composition fragments, reflect the superposition of possible perceptions and interpretations of a piece of music. This paper concentrates on these working analytical stages of MIDIME. It also outlines plans for using the genetic algorithm to develop improvisational agents in the final pipeline stage. 1. MIDIME SOFTWARE PIPELINE MIDIME is a software architecture that distills the structure of finger-picked string music at several levels of abstraction. It accepts MIDI input from a guitar, analyzes playing to determine low level (finger patterns, meter, accents, tempo, root, scale and chords) and high level (composition fragments) musical intent, and generates accompaniment MIDI streams for one or more synthesizers. This paper is about the architecture and algorithms of MIDIME’s fully working analysis stages. Figure 1 illustrates the MIDIME pipeline. MIDI input through Stage 3 and MIDI output are in full working form. Stage 4 is in a working prototype state. Each stage in Figure 1 is a software thread that analyzes data from the preceding stages and writes its interpretations to its output data table. A data table is a memory resident data structure which its writer updates and which downstream writers read. Each data table is a first-in first-out circular queue of rows owned by its writer. A writer controls its data table by locking the single data row that it is working on until it makes a change that it must expose to downstream readers; the writer then releases the lock, locks the following row, and copies the completed row into the new row, where it repeats the cycle. This is a conventional queuing architecture, with the novelty that reader threads entail the overhead of blocking on a lock only when polling determines that they need access to the row under construction. The contents of a data table depend on its pipeline stage. A row in a Stage 1 data table contains guitar string state for a given time period. A Stage 2 row contains musical structure data for a given time period. A Stage 3 row contains indices showing where the current performance matches the state of a composition map Incoming MIDI messages from guitar. Stage 4 agents generate MIDI accompaniment streams. Outgoing MIDI to software or hardware synthesizer. Stage 2 extracts rhythm, scales, chords, drones & melody. Stage 3 matches Stage 2 output to a practice derived score map of scale-chord-drone-melody-time tuples. Stage 1 maps MIDI stream to state of guitar strings. Figure 1: MIDIME pipeline architecture. derived from practice performances. A Stage 4 row contains outgoing MIDI messages. Each writer is alternately a reader that waits for new data from upstream writers, and a writer that emits another row to downstream readers. Overall, the pipeline of Figure 1 acts as a reactive system that responds to incoming messages within application time constraints. Each writer has the job of waiting for data tailored to its application requirements within the pipeline, transforming the data to meet the requirements of the next downstream writer, and releasing that data. Analysis steps are sometimes complex, but they are always restricted to a tractable single level of abstraction. 2. GUITAR STRINGS OVER TIME Stage 1 converts a stream of finger-picked MIDI messages to a two dimensional matrix of guitar string state over time, where the most recent row is the current state of the strings, and each column is a string. Writer constructor parameters establish the number of strings, MIDI channel-to-string mappings, and minimum noteon velocity and duration. Stage 1 discards any note with velocity or duration below its minimum threshold as a transient. The main jobs of the Stage 1 writer are to mapMIDI channelsto-string positions, map pitch bend messages to discrete semitones while discarding extraneous pitch bends, record string / note state in a Stage 1 row, and release that row to downstream readers. The guitar synth uses one MIDI channel per string. Table 1 shows a short trace of the state of a pluck on the E note at the second fret of the fourth string, followed by a slide to the F at the third fret. The left column shows incoming MIDI messages and the right column shows Stage 1 data rows as they appear in the MIDIME debug graphical user interface. A MIDI trace includes buffer row number, message type, channel, velocity for noteon and